ILR-Based MT Comprehension Test with Multi-Level Questions
نویسندگان
چکیده
We present results from a new Interagency Language Roundtable (ILR) based comprehension test. This new test design presents questions at multiple ILR difficulty levels within each document. We incorporated Arabic machine translation (MT) output from three independent research sites, arbitrarily merging these materials into one MT condition. We contrast the MT condition, for both text and audio data types, with high quality human reference Gold Standard (GS) translations. Overall, subjects achieved 95% comprehension for GS and 74% for MT, across all genres and difficulty levels. Interestingly, comprehension rates do not correlate highly with translation error rates, suggesting that we are measuring an additional dimension of MT quality.
منابع مشابه
A Reading Comprehension Corpus for Machine Translation Evaluation
Effectively assessing Natural Language Processing output tasks is a challenge for research in the area. In the case of Machine Translation (MT), automatic metrics are usually preferred over human evaluation, given time and budget constraints. However, traditional automatic metrics (such as BLEU) are not reliable for absolute quality assessment of documents, often producing similar scores for do...
متن کاملThe Effect of Text Difficulty on Machine Translation Performance -- A Pilot Study with ILR-Rated Texts in Spanish, Farsi, Arabic, Russian and Korean
We report on initial experiments that examine the relationship between automated measures of machine translation performance (Doddington, 2003, and Papineni et al. 2001) and the Interagency Language Roundtable (ILR) scale of language proficiency/difficulty that has been in standard use for U.S. government language training and assessment for the past several decades (Child, Clifford and Lowe 19...
متن کاملساخت و هنجاریابی آزمون تشخیصی سطح خواندن برای دانشآموزان پایه سوم دبستان
AbstractIntroduction: Early diagnosis of school children with dyslexia has an important role in the pre- vention of its harmful consequences. This research was carried out with the aim of construc-tion, standardization and the assessment of the validity and reliability of reading level diagnostic test for third grade primary school children in the city of Isfahan.Method: Five hundred sixty nin...
متن کاملThe Effect of Iranian EFL Learners’ Self-generated vs. Group-generated Text-based Questions on their Reading Comprehension
Reading comprehension is one of the most important skills, especially in the EFL context. One way to improve reading comprehension is through strategy use. The present study aimed at investigating the effect of question-generation strategy on learners' reading comprehension. The participants in the study were 63 intermediate students from three intact groups in Resa institute in Boukan, They we...
متن کاملEffects of Closed-caption Programs on EFL Learners’ Listening Comprehension and Vocabulary Learning
This study aimed at investigating the impact of closed-caption program on listening comprehension of English movies and vocabulary learning. Sixty-four graduate students studying at Shiraz Islamic Azad University were selected as the participants of the study. The participants were divided into two groups: experimental group (with closed caption program) and control group (without closed captio...
متن کامل